appreciate if someone can review this PR.

Hi @snnn , would you be able to review and provide feedback on this PR? appreciate your time.

Hi, I have rebased the PR to resolve the merge conflicts. I'm happy to address any feedback you may have. Thank you!

I have checked out the changes and run performance test and accuracy tests with and without flag using onnxruntime_perf_test (modified the binary to dump output for comparisons) on AWS Graviton3 instances and it was fine.

Hi @chenfucn , @yufenglee , I have updated the PR (1) to move to the newer gemm interface and (2) to add session option based fastmath mode control. Please review and let me know your feedback.

Hi @chenfucn , @yufengle, appreciate if someone can trigger the CI for this PR. I have addressed all the feedback except the windows testing for which I'm waiting for the Windows CI results. Thank you!

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline

/azp run ONNX Runtime Web CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

Thanks for the review, I will update the PR to address this and also add unit tests.

I have updated the PR to address all the feedback so far and also the learnings from my other qgemm PR.
(1) added the feature only for not Apple
(2) added mlas unit tests
(3) tested linux full build (both release and release with debug info)
(4) minimal build
(5) android build with cross compilation on x86. and (5) lintrunner and git-clang-format

Next, adding ort optimizer and provider tests to test the fastmath session.
Please review and let me know if any feedback on this version.

@snadampal did you push your change to Github?

not yet,planning to push the code format changes along with the session name change

Hi @skottmckay , appreciate your response on this.

Hi @skottmckay , if you could comment on the naming part, I will update the PR. the following is what I think should cover your suggestion. You had suggested replacing session with mlas. I think session makes sense because the config is still session specific, so I left it, but added mlas to it.

static const char* const kOrtSessionOptionsMlasGemmFastMathMode = "session.enable_mlas_gemm_fastmath_mode";

Hi @skottmckay Scott McKay, appreciate your response on this.

Hi @skottmckay

        Scott McKay
        , if you could comment on the naming part, I will update the PR. the following is what I think should cover your suggestion. You had suggested replacing session with mlas. I think session makes sense because the config is still session specific, so I left it, but added mlas to it.

static const char* const kOrtSessionOptionsMlasGemmFastMathMode = "session.enable_mlas_gemm_fastmath_mode";

I think I would consider the first name as something that points me to where I would find the setting being used. e.g. 'optimization' means look in the optimizer project. I would say it's inferred you're configuring something in the session as you're using SessionOptions (vs. say RunOptions). Based on that, I would vote for 'mlas.' as the prefix.

The name also seems a little too generic as it sounds like it would apply to MLAS as a whole. Unless we think there will be some other fastpath that applies to MLAS GEMM in general, a more specific name would be clearer. e.g. mlas.enable_gemm_fastmath_arm64_bfloat16

Or alternatively the platform/datatype could be in the value and you could parse that.

e.g. mlas.enable_gemm_fastmath_mode could have a value of arm64.bfloat16 and additional platform.data_type values could be parsed for. That is obviously more complicated so we should avoid unless we think it would be used. Possibly required if we had this GEMM fastpath on multiple platforms or for multiple data types though as I assume I'd want to be able to enable/disable each specific combination, and having a new config key for every single combination doesn't scale well.

thank you, I see your point. bf16 and f16 are the potential fastmath options, but on aarch64, so far I see interest for bf16 fastmath alone. I agree that there may not be multiple of these for different platforms, so I will go ahead with a simple config key.

static const char* const kOrtSessionOptionsMlasGemmFastMathArm64Bfloat16 = "mlas.enable_gemm_fastmath_arm64_bfloat16";

Update the PR for the session name and other points discussed so far including the clang-formatting. Tested

1. release, debug and minimal builds on aarch64 neoverse v1 and n1 platforms
2. android build and linux cross compilation for aarch64 config on x86 platform

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

Thanks to @chenfucn , @snnn , @skottmckay and @yufenglee for the great feedback and merging the PR!

@snadampal , thanks for making ONNX Runtime better. Welcome to bring more changes to us. You have my email. Do not hesitate to contact me anytime when you need help on reviewing PRs.